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ABSTRACT 



Background: Citation analysis as one of the most widely used methods of bibliometrics 
can be used for computing the various impact measures for scholars based on data from 
citation databases. Journal Citation Reports (JCR) from Thomson Reuters provides an- 
nual report in the form of impact factor (IF) for each journal. 

Objectives: We aimed to evaluate the citation parameters of Hepatitis Monthly by JCR in 
2010 and compare them with GS and Sc. 

Materials and Methods: All articles of Hepat Mon published in 2009 and 2008 which 
had been cited in 2010 in three databases including WoS, Sc and GS gathered in a spread- 
sheet. The IFs were manually calculated. 

Results: Among the 104 total published articles the accuracy rates of GS and Sc in record- 
ing the total number of articles was 96% and 87.5%. There was a difference between IFs 
among the three databases (0.793 in ISI [Institute for Scientific Information], 0.945 in Sc 
and 0.85 GS). The missing rate of citations in ISI was 4% totally. Original articles were the 
main cited types, whereas, guidelines and clinical challenges were the least ones. 
Conclusions: None of the three databases succeed to record all articles published in the 
journal. Despite high sensitivity of GS comparing to Sc, it cannot be a reliable source for 
indexing since GS has lack of screening in the data collection and low specificity. Using 
an average of three IFs is suggested to find the correct IF. Editors should be more aware 
on the role of original articles in increasing IF and the potential efficacy of review articles 
in long term impact factor. Published by Kowsar Corp, 2012. cc 3.0. 



► Implication for health policy/practice/research/medical education: 

Impact factor reported by JCR measures the evaluation of both journal and article quality. In this study we compare citation report 
by three main databases (ISI, Scopus, and Google Scholar). All researchers who are keen on the field of medical journalism and IF 
evaluation and articles' quality measurement are recommended to study this article. 

► Please cite this paper as: 
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1. Background 

The journal of "Hepatitis Monthly" (Hepat Mon) as one 
of the leading journals in the field of hepatology in the 
region has been published since 2001. The Journal con- 



text is devoted to particular compilation of the latest 
worldwide and interdisciplinary approach and findings 
including original manuscripts, meta-analyses, review 
articles, health economic papers, debates and consensus 
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statements of clinical relevance of hepatological field es- 
pecially liver diseases. By the end of 2010, more than 220 
papers have been published in 28 issues from them 210 
articles are indexed in ISI (Institute for Scientific Informa- 
tion) Web of Science since 2007. Full-text of articles can 
be found in many indexing sources like Scopus and EM- 
BASE which allows the wide exposure and opportunity 
for sound and insightful content to be cited by other re- 
searchers throughout the world. Citation analysis, as the 
examination of the frequency, patterns, and graphs of 
citations in articles and books, is one of the most widely 
used methods of bibliometrics and is suitable applica- 
tion for the identification of expert referees to review pa- 
pers, to provide transparent data in support of academic 
merit review, tenure, and promotion decisions. The cita- 
tion analysis tools can be used for computing the various 
impact measures for scholars based on data from citation 
indices. 

Web of Science (WoS) is an online academic citation in- 
dexing source provided by Thomson Reuters since 1900. 
Journal Citation Reports (JCR) is a comprehensive re- 
source that allows readers to evaluate and compare jour- 
nals using citation data drawn from overll,000 scholarly 
and technical journals of more than 3,300 publishers in 
over 80 countries. JCR shows the most frequently cited 
journals and the highest impact factor (IF) of the jour- 
nals in a field. Google Scholar (GS) is a freely accessible 
web search engine that indexes the full text of scholarly 
literature across an array of scholarly publishing formats 
either from peer-reviewed or non-peer-reviewed journals 
since 2004. Scopus (Sc), officially named SciVerse Scopus, 
is a bibliographic database containing abstracts and ci- 
tations for academic journal articles which covers nearly 
18,000 titles from over 5,000 international publishers, 
including coverage of 16,500 peer-reviewed journals in 
the scientific, technical, medical, and social sciences (in- 
cluding arts and humanities). It is owned by Elsevier and 
is available online by subscription. 

2. Objectives 

We aimed to evaluate the accuracy of Hepatitis Monthly 
IF presented by JCR in 2010 comparing with GS and Sc and 
evaluate the advantages and threats of JCR in comparison 
with GS and Sc. 

3. Materials and Methods 

Based on JCR 2010, we selected all articles of Hepat Mon 
published in 2009 and 2008 which had been cited in 
2010. Using JCR 2010, Sc and GS report, we gathered all 
data in a spreadsheet then cleared dataset and prepared 
them for analysis. Calculation of the IF was done based on 
the JCR method provided by Thomson Reuters (Table I ). 
The IF from JCR was assumed as the gold standard when 
calculating others from Sc and GS. 



3.1. Method of data collection and analysis 

For the citation indicator of Hepatitis Monthly, the 
published articles during 2008 and 2009 have been con- 
sidered for searching in three databases: ISI, Sc, and GS. 
Each article searched separately in the databases and all 
citations recorded in a spreadsheet. Afterwards IFs were 
manually calculated from the spreadsheet which can be 
found in Table 2. To calculate IF of Hepat Mon in WoS, it 
was searched through the Cited Reference Search in WoS 
as well as the final report of JCR, 2010. According to its 
policy, ISI don't count the letters to the editors and edito- 
rial articles in IF calculation. Notably ISI was considered 
as the gold standard for calculations. 

4. Results 

Among 104 total published articles in Hepatitis Month- 
ly during 2008 and 2009, 87, 100, and 91 articles were 
recorded in ISI, GS, SC respectively (Table 2). Due to omis- 
sion of "letters to editor" and "editorials" in calculation 
of IFs of ISI, the real number of published articles were 
different from those indexed in ISI. Consequently and 
thanks to assumption of ISI as the gold standard, the ac- 
curacy rates of GS and Sc in recording the total number 
of articles were 96% and 87.5%, respectively. There was a 
difference between IF among three databases. As shown 
in Table 2 the IF of Hepat Mon in 2010 was 0.793 in ISI, 
while it was 0.945 and 0.85 in SC and GS. There were 72 ci- 
tations in ISI in 2010 to the published articles in 2008 and 
2009, while according to JCR, total number of citations in 
2010 to the published articles in the same year (2008 and 
2009) were 69. Therefore, the missing rate of citations in 
ISI was 4% totally (Table 2). 

Among all published articles, two systematic reviews 
from Iran had the most citations which one of them was 
published in 2008 (n = 9) and another in 2009 (n = 7). Even 
though, there was a difference between the calculated IF 
in GS and Sc with ISI, Table 3 demonstrated that the men- 
tioned difference was shown only in a few articles. The 
main discrepancy was in one original article published 
in 2009. According to data from GS this article is cited 5 
times comparing to ISI and Sc which was two times. Fig- 
ure 1 shows that among articles published in Hepat Mon 
during 2008 and 2009, original articles were the main 
cited types while guidelines and clinical challenges were 
the least types. Whereas if we consider proportion of cita- 
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Table 2. Total Number of Articles, Citations and Self-Citations of Two Volumes (2008 and 2009) of Hepatitis Monthly in Googl 
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a The results in this section is based on JCR 

b The actual citation number based on web of science were 71 citations (missing rate in comparison with JCR calculation: 4%) 



tions to the types of articles, review articles have the most 
citation. Furthermore, case reports were the only type of 
articles which were the same in citation numbers in all 
the databases (Figure 1). No significant difference was de- 
tected in all other types of articles except original articles 
which represented a significant difference between ISI 
and the two other databases. 

Based on JCR, out of all citations to Hepatitis Monthly 
in 2010, "IRAN RED CRESCENT MED" had the most cita- 
tions (n = 24) and then "ASIAN PAC ] CANCERP" (n = 5) and 
"ARCH MED RES" (n = 4) were in the second and third lev- 
els (Table 4). Moreover, the most citations to Hepat Mon 



were from Iran (15.78%), Japan (3.28%) and USA (2.63%). 
Analyzing IF of the journals that cited to Hepat Mon 
doesn't show any relationship between the journals IF 
and number of citation to Hepat Mon (the average IF of 
1.86 for the first 10 journals compare to 2.139 for the sec- 
ond 10 journals). Among all citations in Hepat Mon, 90 
citations belonged to "HEPATOLOGY" (IF: 10.885), 50 to "J 
HEPATOL" (IF = 9.334), 36 to "GASTROENTEROLOGY" (IF = 
12.032), and 36 to "NEW ENGL J MED" (IF = 53.486). There- 
fore, there was no correlation between IF and number of 
citations in Hepat Mon (Table 5). Generally, the average IF 
of the first 10 journals is 10.517 and the average IF for the 
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Table 4. Distribution of the Top 20 Journals Which Cited in 2010 to Published Articles of Hepatitis Monthly (Data Gathered From JCR, ISI, 2010) 



Impact Factor 
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Table 5. Distribution of the Top 20 Journals Which Are Cited 


in Hepatitis Monthly in 2010 (Data Gathered From JCR, ISI, 2010) 


Impact Factor 
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Figure 1. Comparison of Number of Citations in ISI, GS, and Sc Based on 
Article Types Published in Hepatitis Monthly (2008, 2009) 

second 10 journals is 9.05. 

5. Discussion 

The accuracy rates of GS and Sc in recording the to- 
tal number of articles were 96% and 87.5%. Although GS 
had indexed more articles than Sc, it is not reasonable to 
rely on it for counting the total number of articles since 
fundamentally GS is a robot which collects data via the 
websites with neither human interaction nor data clean- 
ing while Sc is guided and supervised by operators, by an- 
other meaning, GS has high sensitivity and low specificity 
vs. Sc with high specificity, while GS reports all citations 
from different types of publications, therefore, it cannot 
be as specific as the two other databases (1). GS have found 
to be sometimes inadequate, and less often updated (2). 
Moreover, Using GS does not guarantee to access the full- 
text availability for the readers since it is regulated via the 
journal's publisher (3). Another important factor is that 
the relative coverage of GS varies by discipline compared 
to other general databases (4). It is also indicated that re- 
liability of each database depends on some factors such 
as year and the subject of publication (5). Despite much 
more reliability and validity of Sc, there are some doubts 
about Sc. Since the Elsevier is the owner of Sc, and is also 
one of the main international publishers of scientific 
journals, some users have misgivings about a potential 
conflict of interest in the choice of the periodicals to be 
included in the database (3). 

There was a difference between IF among three databas- 
es. (GS = 0.85, Sc = 0.945). Notably Sc covers more interna- 
tional and open access journals than the other databases 
(5) and this could be the potential reason for higher IF 
among all. Although we found difference among those 
three databases, Bauer et al. in 2005 showed no signifi- 
cant difference between results by Sc and WoS (1). Just 
similar to Bauer (1), Bakkalbasi et al. in 2006 reported dif- 
ferences in citations of all pairs databases except GS and 
Sc (5). In contrast with previous studies by Bauer and Bak- 
kalbasi and like to ours, another study revealed higher 
coverage of articles by Sc than WoS (6). "Cited by" feature 



in GS, found to be inadequate and less often updated in 
comparison to the other two databases (2). Respect to 
the nature of GS which is a crawler, a significant problem 
with GS is the secrecy about its coverage because some 
publishers do not allow it to crawl their journals. For 
instance Elsevier journals were not included before mid- 
2007, when Elsevier began to make most of its ScienceDi- 
rect content available to GS and Google's web search (1, 7). 
List of journals crawled are not known in GS as well as the 
frequency of its updates (2 ). Copyright issues in accessing 
to the contents of the most expensive commercial data- 
bases are still discussed between Google and the publish- 
ers like Thomson Rheuters. GS crawlers collect blindly 
citations and manipulated hitting in a webpage raises its 
ranking consequently brings the page to the tops of the 
search results. This false effect of GS on citation counts is 
criticized with many searchers. Therefore citation counts 
from GS are not a reliable metrics for counting the h-in- 
dex and impact factor. 

Another issue for searchers in GS is searching with In- 
terpunctuation characters in titles which produce wrong 
search results, and authors are assigned to wrong papers, 
which leads to erroneous additional search results. Some 
editors prefer to negotiate with ISI to recategories the 
published articles to be considered as citable articles (8). 
Bagatin et al. believe IF cannot completely represent qual- 
ity of articles, since review articles have a high potential 
for citations while the publishing case reports usually 
have a negative effect on citation on account of few cita- 
tions (9). Based on the types of articles published in Hepat 
Mon during 2008 and 2009, no significant difference was 
detected in all other types of articles except original ar- 
ticles which represented a significant difference between 
ISI and the two other databases. Review articles receive 
extremely high citations and review journals have some 
of the highest impact factors (10). There is a skewed dis- 
tribution of citations in most fields. The so-called 80/20 
phenomenon applies, in that 20% of articles may account 
for 80% of the citations (11). Publishing mediocre review 
papers will not necessarily boost a journal's impact (11). 

As the free access of PubMed and GS comparing with 
WoS and Sc absorbs more readers and citations while Sc 
offers about 20% more coverage than WoS, whereas GS of- 
fers results of inconsistent accuracy. Till now PubMed re- 
mains an optimal tool in biomedical electronic research. 
Currently Sc is limited to the recent articles that have 
been published since 1995 following which it cannot over- 
come WoS, in spite of wider journal coverage (2). Howev- 
er, some studies emphasize that Sc is more user friendly 
with shorter time span (12). Thanks to alerting features 
via Sc which allows authors to track all published articles, 
it also offers authors a profile which covers affiliations, 
number of publications and their bibliographic data, 
references, and details on the number of citations each 
published document has received. H-index then will be 
calculated in each profile (12). Garfield believes that 24% 
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of references contain only 25 journals (little more than 
1 percent of SCI coverage) (13). This analysis gives good 
reason for concern about any increase in the number of 
scientific and technical journals. Increased number of 
journals makes coverage of the literature more difficult, 
but it seems that most of the current journals play only a 
marginal role, if any, in the effective transfer of scientific 
information (13). Of 2200 journals covered by the SCI in 
1969, 500 journals publish about 70% of all published ar- 
ticles (13). 

None of the three databases succeed to record all ar- 
ticles published in the journal. Despite high sensitivity 
of GS comparing to Sc, it cannot be a reliable source for 
indexing since GS has lack of screening in the data col- 
lection and low specificity. According to the highest IF in 
Sc comparing to GS and ISI it is suggested using an aver- 
age of three IFs to find the correct one. Editors should be 
more aware on the role of original articles in increasing 
IF and the potential efficacy of review articles in long 
term impact factor. 
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